Evaluation of Narrative and Expository Text Summaries Using Latent Semantic Analysis

In this chapter I approach three automatic methods for the evaluation of summaries from narrative and expository texts in Spanish. The task consisted of correlating the evaluation made by three raters for 373 summaries with results provided by latent semantic analysis. Scores assigned by latent semantic analysis were obtained by means of the following three methods: 1) Comparison of summaries with the source text, 2) Comparison of summaries with a summary approved by consensus, and 3) Comparison of summaries with three summaries constructed by three language teachers. The most relevant results are a) a high positive correlation between the evaluation made by the raters (r= 0.642); b) a high positive correlation between the computer methods (r= 0.810); and c) a moderate-high positive correlation between the evaluations of raters and the second and third LSA methods (r= 0.585 and 0,604), in summaries from narrative texts. Both methods did not differ significantly in statistical terms from the correlation among raters when the texts evaluated were predominantly narrative. These results allow us to assert that at least two holistic LSA-based methods are useful for assessing reading comprehension of narrative texts written in Spanish.